Variable imaging arrangements and methods therefor

ABSTRACT

Various approaches to imaging involve selecting directional and spatial resolution. According to an example embodiment, images are computed using an imaging arrangement to facilitate selective directional and spatial aspects of the detection and processing of light data. Light passed through a main lens is directed to photosensors via a plurality of microlenses. The separation between the microlenses and photosensors is set to facilitate directional and/or spatial resolution in recorded light data, and facilitating refocusing power and/or image resolution in images computed from the recorded light data. In one implementation, the separation is varied between zero and one focal length of the microlenses to respectively facilitate spatial and directional resolution (with increasing directional resolution, hence refocusing power, as the separation approaches one focal length).

RELATED PATENT DOCUMENTS

This patent application is a continuation of U.S. patent applicationSer. No. 14/219,896, filed filed Mar. 19, 2014. U.S. patent applicationSer. No. 14/219,896 is a continuation of U.S. patent application Ser.No. 13/542,544, filed Jul. 5, 2012. Patent application Ser. No.13/542,544 is a continuation of U.S. Patent application Ser. No.12/278,714, filed Oct. 9, 2009 (now U.S. Pat. No. 8,248,515). U.S.patent application Ser. No. 12/278,714 is a national stage filing ofInternational Application No. PCI/US2007/003346 filed on Feb. 6, 2007,and claims the benefit, under 35 U.S.C. §119(e), of U.S. ProvisionalPatent Application No. 60/765,903, entitled “Imaging Arrangements andMethods Therefor” and filed on Feb. 7, 2006; this patent document alsoclaims the benefit, under 35 U.S.C. §119(e), of U.S. Provisional PatentApplication No. 60/810,432, entitled “Digital Light Field Photography”and filed on Jun. 2, 2006. This patent document is also acontinuation-in-part of PCT Patent Application No. PCT/US2005/035189(publication number WO/2006/039486), entitled “Imaging Arrangements andMethods Therefor” and filed on Sep. 30, 2005, to which priority isclaimed under 35 U.S.C. §120 for common subject matter. All of theaforementioned patent applications are hereby incorporated by referencein their entirety.

FEDERALLY-SPONSORED RESEARCH AND DEVELOPMENT

This invention was made with Government support under contract 0085864awarded by National Science Foundation. The Government has certainrights in this invention.

FIELD OF THE INVENTION

The present invention relates generally to imaging applications, andmore specifically to an arrangement and approach to facilitate lightfield imaging which accounts for spatial and directional resolution.

BACKGROUND

Imaging applications such as those involving cameras, video cameras,microscopes and telescopes have generally been limited in their abilityto collect light and to process light to generate images havingsignificant depth of field. For example, most imaging devices do notrecord most of the information about the distribution of light enteringthe device, and are limited in their ability to generate focused imageswith a significant depth of field. Conventional cameras such as digitalstill cameras and video cameras do not record most of the informationabout the light distribution entering from the world. In these devices,collected light is often not amenable to manipulation for a variety ofapproaches, such as for focusing at different depths (distances from theimaging device), correcting for lens aberrations or manipulating theviewing position.

The general inability to generate images with significant depth of fieldapplies to both still and video imaging applications. For still-imagingapplications, typical imaging devices capturing a particular scenegenerally focus upon a subject or object in the scene, with other partsof the scene left out of focus. Similar problems prevail forvideo-imaging applications, with a collection of images used in videoapplications failing to capture scenes in focus.

One general approach to solving this problem has been the development ofintegrated light field cameras, which detect the light traveling eachray flowing inside the body of a conventional camera (the “light field”flowing inside the camera body). For example, the plenoptic camerautilizes a microlens array in front of the sensor in order to capturenot just how much light accumulates at each spatial location on theimaging plane, but how much light arrives from each direction.Processing the recorded light rays in different ways allows, forexample, computational refocusing of final photographs, or computationalextension of the depth of field.

As many imaging applications suffer from aberrations with equipment(e.g., lenses) used to collect light, correcting for these aberrationsis desirable and often necessary to generate high quality images. Suchaberrations may include, for example, spherical aberration, chromaticaberration, distortion, curvature of the light field, obliqueastigmatism and coma. Correction for aberrations has historicallyinvolved the use of corrective optics that tend to add bulk, expense andweight to imaging devices. In some applications benefiting fromsmall-scale optics, such as camera phones and security cameras, thephysical limitations associated with the applications make itundesirable to include additional optics. Moreover, as the number ofphotosensors used to collect image data increases, and as thearrangement and processing of data from the same becomes increasinglyimportant, aberration and other conditions that raise issue with thecreation of images can significantly hinder the ability to createaccurate images. Aberrations can particularly inhibit the ability tocreate accurate images focused at different depths.

Difficulties associated with the above have presented challenges toimaging applications, including those involving the collection andprocessing of digital light data for digital imaging.

SUMMARY

The present invention is directed to overcoming the above-mentionedchallenges and others related to imaging devices and theirimplementations. The present invention is exemplified in a number ofimplementations and applications, some of which are summarized below.

According to an example embodiment of the present invention, therelative positioning of photosensors in an imaging arrangement is variedto facilitate a desired trade-off between the spatial and angularresolution of the recorded light data. The recorded light data is usedin conjunction with information characterizing the imaging arrangement(including optical design and configuration of the photosensors) inorder to generate a virtual image in which a portion of the image hasbeen refocused and/or had its image quality corrected. For example, theimage quality may have been corrected by computationally reducing theeffect of optical aberrations due to the optics in the recording device.

According to an example embodiment of the present invention, therelative positioning of photosensors in an imaging arrangement is variedto facilitate the selective plenoptic collection of image data. Light isdetected and used together with directional information characterizingthe detected light, information characterizing the imaging arrangementand the relative positioning of the photosensors to generate a virtualimage that corresponds to a refocused image and, in some instances, animage that is also corrected.

According to another example embodiment of the present invention, adigital imaging system processes data useful to synthesize an image of ascene. The system includes a photosensor array, and an opticsarrangement including a main lens and a microlens array at atwo-dimensional focal plane of the main lens. The optics arrangementdirects light from the scene to the photosensor array via the main lensand microlens array to the photosensor array. A control arrangement setsa relative spatial relationship between the photosensor array and themicrolens array, typically altering their separation while keeping themparallel. For each relative positioning of the photosensor array and themicrolens, each photosensor senses a different set of rays flowingwithin the recording device. This set of rays is a function of theposition of the photosensor array relative to the microlens array, andis used in order to interpret the photosensor values in subsequentprocessing. A processor provides image data characterizing a synthesizedimage as a function of the light sensed at different photosensors, theposition of the photosensors relative to the microlens array, and theoptical configuration of the microlenses and the rest of the opticaltrain in the recording device.

According to another example embodiment of the present invention, adigital imaging system synthesizes an image from a set of light rays.The system includes a main lens, a photosensor array to collect a set oflight rays and to output light data characterizing the set of lightrays, and a microlens array, between the main lens and the photosensorarray, to direct the set of light rays from the main lens to thephotosensor array. A positioning arrangement sets a relative separationbetween the photosensor array and microlens array to facilitateselective detection of directional characteristics of the light ray datarecorded by the photosensor array. In a more particular embodiment, thisselectivity is achieved via a selectable trade-off between the spatialand directional resolution of the recorded light ray data. An image dataprocessor computes a synthesized refocused image, using the light raydata, as a function of the optical properties, physical arrangement andrelative position of the main lens, microlens array and photosensorarray. The synthesized refocused image exhibits a virtual redirection ofthe set of light rays as collected by the photosensor array.

According to another example embodiment of the present invention, acamera having a main lens, a microlens array and a photosensor array isused for digital imaging. The photosensor array is arranged to senselight passed via the main lens to the microlens array and dispersed bythe microlens array. The configuration of the camera is set by settingthe separation between the planes in which the microlens array andphotosensor array lie. The microlens array is used to direct light tothe photosensor array, with the photosensor array including, for eachmicrolens in the microlens array, a plurality of photosensors arrangedto receive light from the microlens. The photosensor array is used tosample the light field flowing within the camera, and the light field iscomputationally resampled as a function of a ray-tracing approach fordetermining where the light rays passing into the camera will terminateon the photosensor array. The digitally resampled light field is used tocompute an output image in which at least a portion of the image isrefocused.

The above summary of the present invention is not intended to describeeach illustrated embodiment or every implementation of the presentinvention. The figures and detailed description that follow moreparticularly exemplify these embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be more completely understood in consideration of thedetailed description of various embodiments of the invention thatfollows in connection with the accompanying drawings, in which:

FIG. 1A is an imaging arrangement for sampling a light field flowingpast a lens, for computing images with refocusing and/or imagecorrection, according to an example embodiment of the present invention;

FIG. 1B illustrates a different set of rays within the imagingarrangement of FIG. 1A, according to another example embodiment of thepresent invention;

FIGS. 1C and 1D show an approach to characterizing detected light raysto facilitate the computation of an image, such as with the arrangementin FIG. 1A, according to another example embodiment of the presentinvention;

FIGS. 2A-4B show an arrangement for sampling light fields for computingimages, with varied microlens-to-photosensor distance in accordance withvarious embodiments of the present invention, wherein

FIGS. 2A-B show a relatively large microlens-to-photosensor distance,

FIGS. 3A-B show an intermediate microlens-to-photosensor distance, and

FIGS. 4A-B show a relatively small (e.g., near-zero)microlens-to-photosensor distance;

FIGS. 5A-5F show an approach to computing an image to facilitatedifferent microlens-to-photosensor distances, according to anotherexample embodiment of the present invention; and

FIGS. 6A and 6B respectively show perspective and cross-sectional viewsof a microlens array and circuit arrangement for computing images, witha microlens-to-photosensor distance actuator, according to anotherexample embodiment of the present invention.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It should be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention

DETAILED DESCRIPTION

The present invention is believed to be useful for a variety of imagingapplications, and the invention has been found to be particularly suitedfor electronic imaging devices and applications involving light fieldsampling and the correction of related images. In these contexts, a“light field” refers to the 4D function that defines the amount of light(e.g., radiance) traveling along each ray in some region of space. Inthe embodiments discussed below the region of space is typically theinterior of the recording optical device, such as the space within thebody of a camera, microscope, telescope, etc. In connection with variousembodiments described below, primary interest is directed to the rays oflight flowing into an imaging plane, such as the focal plane typicallydefined by the photosensor array in a conventional digital camera. Withrespect to this imaging plane, “spatial resolution” refers to thesampling density within the 2D imaging plane itself and “directionalresolution” refers to the sampling density in the 2D angular domain ofrays incident on the imaging plane. While the present invention is notnecessarily limited to such applications, aspects of the invention maybe appreciated through a discussion of various examples using thesecontexts.

According to an example embodiment, an imaging arrangement is adapted toselect a desired spatial and directional resolution in recorded lightfield data to facilitate different performance characteristics in imagescomputed from the detected light. In some applications, the imagingarrangement facilitates greater directional functionality, with a mainlens passing light through microlenses that are focused on the mainlens, to photosensors at or near the focal plane of the microlenses. Inother applications, the aforesaid imaging arrangement controls thepositioning of the photosensors to facilitate greater spatial resolutionand less directional resolution by positioning the photosensors closerto the microlenses (physically and/or optically), relative to themicrolenses' focal plane. In still other applications, the imagingarrangement positions the photosensors immediately adjacent to themicrolenses to facilitate maximum spatial resolution. Characteristics ofthe optical arrangement, including the separation of the microlenses andphotosensors, are used with light ray data from the photosensors tocompute an image that is refocused and/or exhibits image enhancementssuch as aberration correction.

Turning now to the Figures, FIG. 1A shows an imaging system 100 thatfacilitates the collection of light and computation of an image using aselectable photosensor positioning approach to set spatial anddirectional resolution, according to another example embodiment of thepresent invention. The imaging system 100 includes an imagingarrangement 170 having a main lens 110, a microlens array 120, aphotosensor array 130 and an actuator 180 for setting the distancebetween the microlens and photosensor arrays. In the context of examplesherein, the microlens array 120 and photosensor array 130 may bereferred to as implementing a light ray sensor. Although FIG. 1Aillustrates a particular main lens 110 (single element) and particularmicrolens array 120, those skilled in the art will recognize that avariety of lenses and/or microlens arrays (currently available ordeveloped in the future) are selectively implemented with a similarapproach by, for example, replacing the shown main lens and/or microlensarray. For instance, the main lens arrangement 210 in FIG. 2A may beimplemented as main lens 110, in connection with various embodiments.

For illustrative purposes, rays of light from a point on a subject 105in an imaged scene are brought to a convergence point on the microlensarray 120, which is located at the optical conjugate distance of thesubject. For this example, the photosensor array 130 is shown located atthe focal plane 131, with the actuator 180 selectively moving thephotosensor array 130 between the shown position and positions closer tothe microlens array 120. However, the photosensor array 130 may belocated at various separations from the microlens array to providedifferent sampling patterns of the light field. The typical separationrange is between the shown location (at the focal length of themicrolens array 120) and immediately adjacent the microlens array 120(i.e., from zero to one focal length of separation). A microlens 122 atthis convergence point separates these rays of light based on thedirection of the light. In configurations where the separation betweenthe microlens array and photosensor array is equal to the focal lengthof the microlenses, the microlens 122 creates a focused image of theaperture of the main lens 110 on the photosensors (positioned as shown)underneath the microlens. Where oriented laterally as shown,“underneath” refers to a position that is optically beyond themicrolens, relative to the main lens. Other positions of the photosensorarray 130 may involve a positioning similar, for example, to those shownin connection with FIG. 2A, FIG. 3A and FIG. 4A and discussed furtherbelow.

Each photosensor in the photosensor array 130 detects light incidentupon it and generates an output that is processed using one or more of avariety of components. The output light ray data are passed to sensordata processing circuitry 140, which uses the data together withpositional information indicating the location of the photosensor array130 relative to the microlens array 120, as well as selective use of theoptical configuration of the remainder of the imaging arrangement 170,to compute image data 150 of a scene (e.g., including subject 105).

The sensor data processing circuitry 140 is implemented, for example,with a computer or other processing circuit selectively implemented in acommon component (e.g., a chip) or in different components. In variousspecific embodiments also according to the present invention, the inputto the sensor data processing circuitry 140 includes the detected lightray values and the geometrical and optical configuration of the imagingarrangement 170. From these inputs, the sensor data processing circuitry140 computes output imagery selectively exhibiting a refocused imageand, where appropriate, a corrected image (where refocusing may includecorrecting). Various approaches to processing detected light data aredescribed in detail herein, including those approaches described above,with and without reference to other figures. These approaches may beselectively implemented with an approach similar to that described withthe sensor data processing circuitry 140.

FIGS. 1C and 1D show an approach to characterizing the light fieldsampling pattern to facilitate the computation of an image, such as withthe arrangement in FIG. 1A, according to other example embodiments ofthe present invention. FIGS. 1C and 1I) and the following descriptionmay also be applied in connection with other approaches describedherein, including those described with FIGS. 2A-4B below. Generally,FIG. 1C shows an optical imaging arrangement 101, similar to that shownin FIG. 1A, with a main lens and photosensor arranged to detect lightfrom a world focal plane and to pass light data to sensor processingcircuitry 141 (similar to circuitry 140 in FIG. 1A) that is adapted tocompute a refocused image using a Cartesian approach.

For illustration, a single ray 102 is characterized as passing through apoint “u” at the main lens 104 to a point “x” on the photosensor(imaging plane) 106. Of course in general the ray exists in threedimensions and we would consider intersections (u, v) at the lens and(x, y) on the imaging plane. Let us refer to the value of the lightfield along the depicted ray as L(x, y, u, v), or L(x, u) if we areconsidering the two-dimensional simplification.

FIG. 1D shows a Cartesian ray-space plot 103 as an abstractrepresentation of the two-dimensional light field, and is used incomputing an image at the sensor processing circuitry 141. The raydepicted on the left is shown as a point (x, u) on the Cartesianray-space in plot 103. In general, each possible ray in FIG. 1Ccorresponds to a different point on the ray-space diagram on FIG. 1D.This example approach to depicting the position related to light data isused in computing an image with various example embodiments below.

Returning to FIG. 1A, as discussed above, the position of thephotosensor array 130 is set, relative to the microlens array 120, tofacilitate certain imaging approaches. As the photosensor array 130 ismoved from the focal plane 131 towards the microlens array 120, theimages under each microlens (referred to as “microlens images”) becomedefocused. Each level of defocus provides a different light fieldsampling pattern, and these have different performance trade-offs (e.g.,between spatial and directional resolution). As shown in FIG. 1A, thephotosensor array 130 is located at approximately the focal plane of themicrolens array 120, such that the microlens images are focused on themain lens. This configuration provides a relative maximal directional“u” resolution and relative minimal “x” spatial resolution on theray-space of the recorded light field.

When the photosensor array 130 is moved to a position between the focalplane 131 and the microlens array 120, horizontal lines in the ray-spacesampling pattern tilt, thereby becoming more concentrated vertically andless so horizontally. This effect can be intuitively understood in termsof the shearing of the light field when focusing at different depths.For illustrative purposes, the focus change can be considered to takeplace in a microscopic camera composed of each microlens and its patchof pixels, and the shearing takes place in the microscopic light fieldinside this microlens camera.

When the photosensor array 130 is located at close to zero separation(i.e., essentially against the microlens array), the imaging system 100performs similarly to a conventional camera. The corresponding ray-spacesampling converges to a pattern of vertical columns as the separationbetween microlenses and photosensor approaches zero. In this case themicrolenses are almost completely defocused, with a relative minimaldirectional resolution and a relative maximal spatial resolution. Thatis, the values read off the photosensor array 130 for such a zeroseparation approach the values that would appear in a conventionalcamera in the absence of the microlens array 120.

In various embodiments, different portions of the imaging system 100 areselectively implemented in a common or separate physical arrangement,depending upon the particular application. For example, when implementedwith certain applications, the microlens array 120, photosensor array130 and actuator 180 are combined into a common image sensor arrangement160. In some applications, the microlens array 120, photosensor array130 and actuator 180 are coupled together on a common chip or othercircuit arrangement. When implemented with a hand-held device such as acamera-like device, the main lens 110, microlens array 120, photosensorarray 130 and actuator 180 are selectively combined into a commonimaging arrangement 170 integrated with the handheld device.Furthermore, certain applications involve the implementation of some orall of the sensor data processing circuitry 140 in a common circuitarrangement with the photosensor array 130 (e.g., on a common chip).

The microlens array 120 and photosensor array 130 are representativelyshown from a two-dimensional perspective with relatively few microlensesand photosensors, but are readily implemented with varied arrangementsof microlenses and photosensors. For instance, the microlens array 120is generally implemented with a multitude (e.g., thousands or millions)of microlenses. The photosensor array 130 generally includes arelatively finer pitch than the microlens array 120, with a plurality ofphotosensors for each microlens in the microlens array 120. In addition,the f-numbers of the microlenses in the microlens array 120 and thef-number of the main lens 110 are generally set such that light passingvia each microlens to the photosensor array does not significantlyoverlap light passed via adjacent microlenses. In some embodiments, thiscondition is achieved by setting the f-number of the lens to be equal orhigher than the f-number of the microlenses.

The actuator 180 is represented generally, with a variety of actuationapproaches readily implemented for moving the photosensor array 130. Forexample, actuators used in conventional cameras to position optics maybe used in a similar manner to position the photosensor array 130. Forexample, the approaches shown in one or more of U.S. Pat. Nos. 7,167,203and 7,164,446 may be implemented for positioning the photosensor arrayin connection with various example embodiments.

In various applications, the main lens 110 is translated along itsoptical axis (as shown in FIG. 1A, in a horizontal direction) to focuson a subject of interest at a desired depth “d” as exemplified betweenthe main lens and an example imaging subject 105. By way of example,light rays from a point on the subject 105 are shown for purposes ofthis discussion. These light rays are brought to a convergence point atmicrolens 122 on the focal plane of the microlens array 120. Themicrolens 122 separates these rays of light based on direction, creatinga focused image of the aperture of the main lens 110 on a set of pixels132 in the array of pixels underneath the microlens.

FIG. 1B illustrates an example approach to separating light rays, suchthat all rays emanating from a point on a main lens 110 and arrivinganywhere on the surface of the same microlens (e.g., 123) are directedby that microlens to converge at the same point on a photosensor (e.g.,133). This approach shown in FIG. 1B may, for example, be implemented inconnection with FIG. 1A (i.e., with the main lens 111 implemented formain lens 110, with microlens array 121 implemented for microlens array120, with photosensor array 131 implemented for photosensor array 135and with the actuator 181 implemented for actuator 180).

The image that forms under a particular microlens in the microlens array121 dictates the directional resolution of the system for that locationon the imaging plane. In some applications, directional resolution isenhanced by facilitating sharp microlens images, with the microlensesfocused on the principal plane of the main lens. In certainapplications, the microlenses are at least two orders of magnitudesmaller than the separation between the microlens array and the mainlens 111. In these applications, the main lens 111 is effectively at themicrolenses' optical infinity; to focus the microlenses, the photosensorarray 135 is located in a plane at the microlenses' focal plane. Tofacilitate increased spatial resolution, the photosensor array 135 ispositioned closer to the microlens array 121.

The microlens array 120 is implemented using one or more of a variety ofmicrolenses and arrangements thereof. In one example embodiment, a planeof microlenses with potentially spatially varying properties isimplemented as the microlens array 120. For example, the microlens arraymay include lenses that are homogeneous and/or inhomogeneous, square inextent or non-square in extent, regularly distributed or non-regularlydistributed, and in a pattern than is repeating or non-repeating, withportions that are optionally masked. The microlenses themselves may beconvex, non-convex, or have an arbitrary profile to effect a desiredphysical direction of light, and may vary in profile from microlens tomicrolens on the plane. Various distributions and lens profiles areselectively combined. These various embodiments provide samplingpatterns that are higher spatially (correspondingly lower angularly) insome regions of the array, and higher angularly (correspondingly lowerspatially) in other regions. One use of such data facilitatesinterpolation to match desired spatial and angular resolution in the 4Dspace.

FIGS. 2A-4B show an arrangement for detecting light for computingimages, each set of figures (i.e., FIGS. 2A&B; FIGS. 3A&B; and FIGS.4A&B) depicting different microlens-to-photosensor distances, asimplemented in accordance with various example embodiments. In each ofFIGS. 2A, 3A and 4A, the imaging arrangement 200 may be implemented in acamera, microscope or other imaging device, such as that shown in FIG.1A and discussed above. That is, the indicated main lenses, microlensarrays, photosensor arrays and actuators are arranged together andcontrolled to detect light ray data for processing locally and/orremotely. The shown rays are those that are integrated by the samephotosensor pixel with different microlens focus for each position ofthe photosensor array. The following more particularly describes eachfigure relative to its indicated microlens-to-photosensor arrayseparation.

Beginning with FIG. 2A, an imaging arrangement 200 includes a main lensarrangement 210 with aperture 212, a microlens array 220 and aphotosensor array 230 that is positioned using an actuator 280 (i.e., totranslate the photosensor array vertically, relative to the positioningshown in FIG. 2A). The actuator 280 has positioned the photosensor array230 at a location relative to the microlens array 220 that is close toor at the focal length of the individual microlenses in the microlensarray, with the microlenses focused on the main lens 210. Thisarrangement facilitates a recorded sampling of the light field withrelatively high directional resolution. By way of example, a set of rays205 is shown in FIG. 2A, and FIG. 2B shows a sampling pattern 240 withthe set of rays integrated by a photosensor pixel 250.

FIG. 3A shows the imaging arrangement 200 with the photosensor array 230positioned by the actuator 280 at an intermediatemicrolens-to-photosensor separation, less than the focal length of themicrolenses in the microlens array 220. With this positioning and asrelative to FIG. 2A, the light field is detected with greater spatialresolution and less directional resolution. A set of rays 305 is shownby way of example in FIG. 3A, and FIG. 3B shows a sampling pattern 340with the set of rays integrated by a photosensor pixel 350. Thehorizontal lines in the sampling pattern 340 are slanted, relative tothose shown in FIG. 2B, with more vertical and less horizontalconcentration.

FIG. 4A shows the imaging arrangement 200 with the photosensor array 230positioned by the actuator 280 at a relatively small (e.g., near-zero)microlens-to-photosensor distance, such that the microlenses in themicrolens array 220 are almost completely defocused. This positioningfacilitates a light field sampling with most if not all of the spatialresolution of the photosensor array 230; however, little to nodirectional information is obtained. A set of rays 405 is shown byexample in FIG. 4A, with a corresponding sampling pattern 440 in FIG. 4Bwith the set of rays integrated by a photosensor pixel 450.

With the above discussion and corresponding figures in mind, thefollowing describes various example embodiments directed to imaging withselective directional resolution as controlled viamicrolens-to-photosensor separation. In certain examples, reference ismade to one or more of the above-discussed figures.

Considering the separation between the microlens array 120 and thephotosensor array 130 in FIG. 1A, each separation of the microlens arrayand photosensor is effectively a different configuration of ageneralized light field camera. By way of example, define β to be theseparation as a fraction of the depth that causes the microlenses in themicrolens array 120 to be focused on the main lens 110. For example, aplenoptic camera configuration corresponds to β=1 (e.g., with themicrolens and photosensor arrays 120 and 130 separated by the focallength of the microlenses), and a configuration involving the microlensand photosensor array pressed up against one another corresponds to β=0.As described above, decreasing the β value defocuses the microlenses byfocusing them beyond the aperture of the main lens 110.

Referring to FIGS. 2A, 3A and 4A, the respective sets of rays 205, 305and 405 respectively reflect an increase in spatial resolution at theexpense of directional resolution as the separation β approaches zero.Reducing the β value results in a shearing of the light field samplingpattern within each column, resulting in an increase in effectivespatial resolution (reduction in x extent), and a corresponding decreasein directional resolution (increase in u extent).

In some applications, higher image resolution is obtained by reducingthe separation between the microlenses and photosensors (by reducingthe/I value) respectively in the microlens and photosensor arrays 120and 130, and by changing the optical focus of the mainlens 110 and finalimage processing (e.g., via sensor data processing circuitry 140). Forinstance, at β=0, the microlenses are pressed up against the sensor andlose their optical power; thus the effective spatial resolution is thatof the sensor. The effective resolution decreases linearly to zero asthe β value increases, with the resolution of the microlens array 120setting a lower bound. In equations, if the resolution of thephotosensor array is M_(sensor)×M_(sensor) (where the photosensor array130 has M×M photosensors) and the resolution of the microlens array isM_(lenslets)×M_(lenslets). (where the microlens array 120 has M×Mmicrolenses) the output images have an effective resolutionM_(effective)×M_(effective), where

M _(effective)=max((1−β)M _(sensor) ,M _(lenslets)).  Equation 1

Changes in the sampling pattern of a generalized light field cameraimplemented in accordance with FIGS. 2A-4D are localized within columnsdefined by the microlens array. Each column represents the microscopicray-space between one microlens and the patch of photosensors that itcovers. Defocusing the microlens by reducing the β value shears themicroscopic ray-space. The example derivation described below inconnection with FIGS. 5A-5E describes an approach to calculating thesampling pattern of the light field as a function of β, for use inconnection with one or more example embodiments of the presentinvention. The derivation begins with the microscopic ray-spaces betweeneach microlens and photosensor, where the sampling pattern is trivial,moving out into the ray-space of a full camera.

FIG. 5A is a schematic for a light field camera 500 having a main lens510, microlens array 520 and photosensor array 530, showing a microlenslabeled i whose microscopic light field is to be analyzed further. FIG.5B illustrates a close-up of the microlens i, with its own localcoordinate system, and a portion of the photosensor array 530 locatedunder the microlens. The microscopic light field's ray-space isparameterized by the intersection of rays with the three illustratedplanes: the microlens plane, x^(i), the sensor plane, s^(i), and thefocal plane of the microlens, w^(i). In order to map this microscopicray-space neatly into a column of the macroscopic ray-space for thewhole camera, the origins of the three planes are chosen to lie alongthe line 505 passing through the center of the main lens 510 and thecenter of microlens i, as indicated on FIGS. 5A and 5B. The focal lengthof the microlenses is f, and the separation between the microlens array520 and the photosensor array 530 is βf, using the abovecharacterization of β.

FIG. 5C illustrates the shape of an example sampling pattern on theray-space parameterized by the microlens x^(i) plane and the photosensors^(i) plane. This choice of ray-space parameterization shows that thesampling is given by a rectilinear grid, since each photosensor pixelintegrates all the rays passing through its extent on s^(i), and theentire surface of the microlens on x^(i). Let us denote the microscopiclight field as l^(i) _(βf)(x^(i), w^(i)) where the subscript βf refersto the separation between the parameterization planes. The samplingpattern in this ray-space is transformed into the macroscopic ray-space,using a change in coordinate systems.

FIG. 5D illustrates a first transformation: re-parameterizing l^(i)_(βf) by changing the lower parameterization plane from the sensor planes^(i) to the microlens focal plane w^(i). The microscopic light fieldparameterized by x^(i) and w^(i) is denoted as l^(i) _(f)(x^(i), w^(i))where the f subscript reflects the increased separation of one microlensfocal length. Reparameterizing into this space introduces shear in thelight field, with the following characterization:

$\begin{matrix}{{l_{\beta \; f}^{i}\left( {x^{i},s^{i}} \right)} = {l_{f}^{i}\left( {x^{i},{{x^{i}\left( {1 - \frac{1}{\beta}} \right)} + \frac{s^{i}}{\beta}}} \right)}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

The transformation from the microscopic light fields under eachmicrolens into the macroscopic ray-space of the camera involves twosteps. First, there is a horizontal shift of Δx^(i), as shown on FIG.5A, to align their origins. The second step is an inversion and scalingin the vertical axis. Since the focal plane of the microlens i isoptically focused on the main lens 510, every ray that passes through agiven point on l^(i) _(f) passes through the same, conjugate point onthe u. The location of this conjugate point is opposite in sign due tooptical inversion (the image of the main lens appears upside down underthe microlens). It is also scaled by a factor of Flf because of opticalmagnification. Combining these transformation steps,

$\begin{matrix}{{l\frac{i}{f}\left( {x^{i},w^{i}} \right)} = {L\left( {{{\Delta \; x^{i}} + x^{i}},{{- \frac{F}{f}}w^{i}}} \right)}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

Combining Equations 2 and 3 gives the complete transformation from l^(i)_(βf) the macroscopic space:

$\begin{matrix}{{l_{\beta \; f}^{i}\left( {x^{i},s^{i}} \right)} = {L\left( {{{\Delta \; x^{i}} + x^{i}},{{x^{i}\frac{F}{f}\left( {\frac{1}{\beta} - 1} \right)} - {\frac{F}{f}\frac{s^{i}}{\beta}}}} \right)}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Equation 4 shows that the slope of the grid cells in FIG. 5E isdiscussed further below.

$\begin{matrix}{{\frac{F}{f}\left( {\frac{1}{\beta} - 1} \right)},} & {{{Equation}\mspace{14mu} 5}\mspace{11mu}}\end{matrix}$

The above discussion holds for the microscopic light fields under eachmicrolens in the microlens array 520. FIG. 5E illustrates thetransformation of all these microscopic light fields into themacroscopic ray space, showing how they pack together to populate theentire space. FIG. SF illustrates that the main lens 510 truncates thesampling pattern vertically to fall within the range of u values passedby the lens aperture.

In the above description, the main lens is assumed ideal, and thef-numbers of the system are matched to prevent cross-talk betweenmicrolenses. In many applications, aberrations affect the above approachand various conditions are addressed via computational approachessimilar to the above. For general information regarding imaging, and forspecific information regarding lens aberration and correction therefor,reference may be made to related International (PCT) Application SerialNo. PCT/US2007/003346 and corresponding PCT Publication No.WO/2007/092545, entitled “Correction of Optical Aberrations”.

In connection with various example embodiments, including thosediscussed in connection with the figures above, the focus of a main lensis set as a condition of the separation between a correspondingmicrolens array and photosensor array. For instance, referring to FIG.1A, the main lens 110 is focused relative to the distance between themicrolens array 120 and the photosensor array 130 (e.g., correspondingto the distance βf as referenced in FIG. 5B).

In one implementation involving intermediate β values, a high finalimage resolution is obtained by optically focusing slightly beyond thesubject (e.g., 105 in FIG. 1A), and use digital refocusing to pull thevirtual focal plane back onto the subject of interest. For example,referring to FIG. 3A, desirable (e.g., maximal) spatial resolution ofresultant photographs is achieved by digitally refocusing slightlycloser than the world focal plane, which is indicated by the upperhorizontal line on the ray-trace diagram (at X). The refocus plane ofgreatest resolution corresponds to the plane passing through the pointwhere the convergence of world rays is most concentrated (where thereference number 305 points in FIG. 3A). Optically focusing furtherfacilitates a shift of this concentrated position of world rays onto thefocal plane. As the β value increases, this concentrated position movescloser to the main lens.

Referring again to FIGS. 5A and 5B, some embodiments use the followingapproach to produce (about) maximum resolution in an output reftocusedimage generated from the generalized light field camera. In thefollowing equations, the assumption is that the desired outputphotograph is focused as if a conventional photosensor were placed at adepth F from the main lens 510. In order to achieve this, thegeneralized light field camera's optical focus is set such that theseparation between the main lens 510 and the microlens array is given by

$\begin{matrix}{F_{opt} = {F + {\frac{\beta}{\left( {\beta - 1} \right)}{f.}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

The maximum-resolution output image is computed by digital refocusing tocompute a virtual image in which the virtual separation between the mainlens 510 and imaging plane is F.

For the range of β values described with FIGS. 2A, 3A and 4A (0<β≦1),F_(opt)<F, meaning that the microlens plane is brought closer to themain lens to optically focus further in the world. The optical mis-focusis the difference between F_(opt) and F, given by

$\begin{matrix}{\frac{\beta}{\left( {\beta - 1} \right)}{f.}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

As β approaches 1, the optical mis-focus asymptotes to negativeinfinity, as the slope of the sampling grid cells becomes toohorizontal, and the optimal resolution is dominated by the verticalcolumns of the sampling pattern set by the resolution of the microlensarray, not by the slope of the cells within each column. When thisoccurs, the optical focal depth is set at the desired focal depth (i.e.,F_(opt)=F), to provide the greatest latitude in refocusing about thatcenter. In some applications, Equation 6 is no longer used once theeffective resolution for the f configuration falls to less than twicethe resolution of the microlens array.

As discussed above, the predicted effective resolutionMeffective×Meffective of the output images is

M _(effective)=max((1−β)M _(sensor) ,M _(lenslets)).  (repeatingEquation 1)

As with F_(opt), the predicted value for M_(effective) derives from ananalysis of the ray-space sampling pattern. Refocusing aligns theimaging projection lines with the slope of the grid cells, facilitatingthe extraction of higher spatial resolution in the sheared samplingpattern. With this approach, the effective resolution of the computedimage is equal to the number of grid cells that intersect the x axis.Within each microscopic light field, Equation 2 shows that the number ofgrid cells crossed is proportional to (1−β), because of the shearing ofthe microscopic light field sampling patterns. Hence, the overallresolution is proportional to (1−β). The maximum possible resolution isthe resolution of the sensor, and the minimum is the resolution of themicrolens array.

In some embodiments, when recording light fields with intermediate β, anauto-focus sensor is used to indicate the depth of the subject. If theauto-focus sensor indicates that the subject is at depth F, the sensoris positioned at F_(opt) as described above. Digital refocusing ontodepth F after exposure is used to produce an image with a large (e.g.,maximum) possible effective resolution ofmax((1−β)M_(sensor),M_(lenslets)).

In connection with various embodiments discussed above involvingrefocusing, output images are anti-aliased by combining samples frommultiple spatial locations. In this regard, images with non-zero opticalmis-focus are used to produce higher-quality images (e.g., relative toimages with zero mis-focus), as facilitated by positioning photosensorsrelative to microlenses in a mis-focused position.

The relative positioning of microlenses and photosensors is chosen tofacilitate the particular application, desirable spatial resolution anddirectional resolution. In one embodiment, the separation betweenmicrolenses and photosensors is set to about one half of the focallength of the microlenses to facilitate spatial resolution with all butabout 2×2 of the full photosensor resolution. For intermediateseparations, the spatial resolution varies continuously between theresolution of the microlens array and that of the photosensor. Therefocusing power decreases roughly in proportion to the increase inspatial resolution, and in some instances, any loss in directionalresolution as the separation between microlenses and photosensors isreduced is well contained within a factor of 2 of an ideal instance.

In one embodiment, a generalized light field camera is adapted forselective variation between a conventional camera with high spatialresolution, and a plenoptic camera with more moderate spatial resolutionbut greater directional resolution and hence refocusing power. Amechanism separating the microlenses and the photosensor is motorizedand controlled via selection to match the needs of a particularexposure. For example, a user could choose high spatial resolution andput the camera on a tripod for a landscape photograph, and later choosemaximal directional resolution to maximize the chance of accuratelyfocusing an action shot in low light.

FIGS. 6A and 6B respectively show perspective “exploded” andcross-sectional views of a microlens array and circuit arrangement 600for computing images, with a microlens-to-photosensor distance actuator652, according to another example embodiment of the present invention.The arrangement 600 may, for example, be implemented in a conventionalcamera, a specialized camera, a microscope arrangement or other videocapture device.

Referring to the exploded view in FIG. 6A, the arrangement 600 includesa digital back 610 that supports a chip package 620 including aphotosensor array 630. A base plate 640 mounts to the photosensor 630. Amicrolens array 660 held by a lens holder 650 is coupled to the baseplate 640, with separation springs 642 separating the base plate andlens holder. An actuator arrangement 652, implemented in threelocations, controls the relative positioning and separation of themicrolens array 660 and the photosensor array 630.

In this example, the resolution of the photosensor array 630 is4096×4096, and the resolution of the microlens array is 296×296(providing about 14×14 sensor pixels under each microlens). Theseparation between an active surface 662 of the microlens array and thephotosensor array 630 is shown at 0.5 mm by way of example, aspositioned by the actuator arrangement 652.

The actuator arrangement 652 includes one or more of a variety ofarrangements, and may be implemented with various arrangements (inposition, type and otherwise) in replacement of that shown in FIGS. 6Aand 6B. One application involves a screw-type fastener with threads thatcontrol the separation of the microlens array 660 and photosensor array630, with the fastener adjusted manually and/or via mechanicalactuation. Other arrangements include those similar to those used inother camera-type motors.

For general information regarding imaging approaches and for specificinformation regarding imaging approaches that may be selectivelyimplemented in connection with one or more various example embodimentsdescribed herein, such as for focusing and/or correcting for lensaberration, reference may be made to PCT Patent Application No.PCT/US2005/035189, entitled “Imaging Arrangements and Methods Therefor,”naming inventors Yi-Ren Ng. Patrick Hanrahan, Marc Levoy, and MarkHorowitz and filed on Sep. 30, 2005, which is fully incorporated hereinby reference.

The various embodiments described above are provided by way ofillustration only and should not be construed to limit the invention.Based on the above discussion and illustrations, those skilled in theart will readily recognize that various modifications and changes may bemade to the present invention without strictly following the exemplaryembodiments and applications illustrated and described herein. Forinstance, such changes may include implementing the various opticalimaging applications and devices in different types of applications,increasing or decreasing the number of rays collected per pixel (orother selected image area), or implementing different algorithms and/orequations than the examples described to assemble or otherwise processimage data. Other changes may involve using coordinate representationsother than or in addition to Cartesian coordinates, such as polarcoordinates, and/or using various other weighting and other schemes tofacilitate the reduction or elimination of aberrations. Suchmodifications and changes do not depart from the true spirit and scopeof the present invention.

What is claimed is:
 1. A digital imaging system for capturing datausable for synthesizing an image of a scene, the system comprising: amain lens positioned to receive light from a scene to be digitallycaptured; a light ray sensor for sensing light from the scene as lightrays of a light field, the light ray sensor comprising, a microlensarray comprising a plurality of microlenses and positioned to receivelight from the scene through the main lens and to capture microlensimages beneath each of the plurality of microlenses; and a photosensorarray comprising a plurality of photosensors, wherein the photosensorarray is positioned to receive light from the microlens array, and in afirst position is coincident with a focal plane of the microlens arraysuch that the microlens images are in focus, and an actuatorconfigurable to set a distance between the photosensor array and themicrolens array, wherein when the distance is shorter than the distanceof the first position, the microlens images become defocused, andwherein each level of defocus provides a difference light field samplingpattern.